AD-A098 809 TENNESSEE TECHNOLOGICAL UNIV COOKEVILLE F/6 9/2 
INTEGRATED MODELING AND ANALYSIS OF FAULT-TOLERANT SYSTEMS WITH=<ETC(U) 
1980 A S COLLINS ApcaR ROOM? 
ee AFOSR“TR=81-0431 


ry ae ey PT yy 


' 1QO < Was Wes 
—— a 
© ge 


Wee 


Was. 


M22 WES Mee 


MICRUCOPY RESOLUTHON TEST CHAR! 


» ~~’ . 


U 


ae osm 81-0481, , . : 


7 LEVEL 


—_— 


S ANTEGRATED MODELING AND ANALYSIS OF | 
CO FAULT-TOLERANT SYSTEMS WITH 7 
GO FAULT-TOLERANT SOFTWARE, 
a : 
ot! WEOSR- El- O17 ap. 
fam ° - 
<=. | 
us ere oe 
Project “No. , PO. No. 80-00721 
F ‘aiid 
7 DTIC 
ELECTE 
MAY 1 2 1981 
Submitted to: E. 


USAF-OSR 
Bolling AFB, D.C. 20332 


Submitted by: 


Aaron S..’Collins, Ph.D 
* Principal Investigator 
Tennessee Technological University 
Cookeville, Tennessee 38501 


ONG FILE copy 


diatributien unlimited. 


81 5 12: Q 16 Approved for publie release} 


—-— = - . + Se ONS te RE aera gl ee —eere - om ge 


TABLE OF CONTENTS 


Page 
LIST“OF- FIGURES “3-8 « % <<. &. + MS cde bs Som Ce Ue iS Ao 1S iii 


| EEOT OP TABLES “3° 4) ww = 2S Gow BE eo hw ae iv 
| INTRODUCTION yo ys! Ye ' p= 4 oe ak BAe Se eee! a db Ge Bee 1 
: OBIECTIVES: 2a wo & GS we Le BEER oe eS 1 
I INTEGRATED HARDWARE/SOFTWARE RELIABILITY MODEL .. . 2 
i] INTEGRATED HARDWARE/SOFTWARE REDUNDANCE . . . . .-. . 9 
i EFFECTS OF INCORPORATING SOFTWARE REDUNDANCY .... 10 
if CONCLUSIONS 2. oo eure Ha eat cae ee ae ee 8 6 
a BIBLIOGRAPHY .. 2... 0. ee ee ee ee ee 1B 


Accession For 


NTIS GRATE 
DTIC TAB [i 
Unannounced 


5: ea enna ena eatom 
Distribution/ a 
Availability Codes 
Avail and/or 
Special 


— 


4iR FORCE uFFICE OF SCIENTIFIC RESEARCH (AFSC) 
BOTICE OF TRANSMITTAL TO DDC 
Tris teehnical report has Leen reviewed amd is 
, SPpreved for public rvlease LAW AFR 190-12 (7d). 
ft Distribution is unlispited, 
- he D. BLOSE 
: - Peehnies) [oferwation Officer 


ii 


me ee a ee et ne ee - eee 
a “ye oe. 


i 
« 
’ 
B 
*” 


FIGURE 


LIST OF FIGURES 


Page 

Failure Density Function for a single Line of 

Computer COGE: «3.<% % /e. s -?  w: S e SCES 
Probability of Failure for upaupi eras 

versus Failures/Line/Hour .. . J &: gt AG 
Probability of Failure for a Single Non-redun- 

dant Module versus Failures/Line/Hour ... . 20 
Probability of Failure for Single TMR Module 

versus Failures/Line/Hour .. . . . ol 
Probability of Failure of coee ee ae 

versus Module Length... . — ee es ee 
Probability of Failure of a ated Module 

versus Module Length .... j 4 wy way fe . oo 
Fortran Program for Calculating Failure Rates 

in Tables 1 through 9 ..........+2. =. 24 
Fortran Program for Calculating Failure Rates 

for Tables 10 through 18 ......... . 25 


1ii 


TABLE 


10 


11 


LIST OF TABLES 


Module and System Failure (1-Reliability) Data 
for 100,000 lines of code divided into 
100,000 modules of 1 line each . 


Module and System Failure (1-Reliability) Data 
for 100,000 lines of code divided into 
10,000 modules of 10 lines each 


Module and System Failure (1-Reliability) Data 
for 100,000 lines of code divided into 4000 
modules of 25 lines each : 


Module and System Failure (1-Reliability) Data 
for 100,000 lines of code divided into 2000 
modules of 50 lines each : 


Module and System Failure (1-Reliability) Data 
for 100,000 lines of code divided into 1000 
modules of 100 lines each se Gaps oie : 


Module and System Failure (1-Reliability) Data 
for 100,000 lines of code divided into 200 
modules of 500 lines each we “S.A 


Module and System Failure (1-Reliability) Data 


for 100,000 lines of code divided into 100 


modules” of 1000 lines each 


Module and System Failure (1-Reliability) Data 
for 100,000 lines of code divided into 20 
modules of 5000 lines each Se eae ke 


Module and System Failure (1-Reliability) Data 
for 100,000 lines of code divided into 10 
modules of 10,000 lines each a a 


Module and System Failure (1~-Reliability) Data 
for 100,000 lines of code divided into 
100,000 modules of 1 line each 


Module and System Failure (1-Reliability) Data 
for 100,000 lines of code divided into 
10,000 modules of 10 lines each : 


iv 


Page 


26 


27 


28 


29 


30 


31 


32 


33 


34 


35 


36 


j 
¢ 
p 
i 
: 
i 
1 


LIST OF TABLES (Cont. ) 


Table 


12 


13 


14 


15 


16 


17 


18 


19 


Module and System Failure (1-Reliability) Data 
for 100,000 lines of code divided into 4000 
modules of 25 lines each 


Module and System Failure (1-Reliability) Data 
for 100,000 lines of code divided into 2000 
modules of 50 lines each ee Be ok 


Module and System Failure (1-Reliability) Data 
for 100,000 lines of code divided into 1000 
modules of 100 lines each 


Module and System Failure (1-Reliability) Data 
for 100,000 lines of code divided into 200 
moduies of 500 lines each . S. se <4 


Module and System Failure (1-Reliability) Data 
for 100,000 lines of code divided into 100 
modules of 1000 lines each 


Module and System Failure (1-Reliability) Data 
for 100,000 lines of code divided into 20 
modules of 5000 lines each ‘ es 


Module and System Failure (1-Reliability) Data 
for 100,000 lines of code divided into 10 
modules of 10,000 lines each 


Variables used in Fortran Programs Listed in 
Figures 7 and 8 


Page 


37 


38 


39 


40 


41 


42 


43 


44 


estetane Melee ~  * 


Introduction 


i 
Research aimed at improving the reliability of computer 


systems has been performed almost independently by a number 
of disciplines in the past. As a result, different modeling 
methods, nomenclature, and perspectives have evolved within 
the various sub-disciplines. More significantly, optimum 
tradeoffs are not achieved during the design process. This 
research effort pulls together results and methodology from 
fault-tolerant architecture and fault-tolerant software to 
determine the effect of including software realities in sys- 
tem structural design decisions for reliability enhancement 
purposes. Specifically, projected software error rate data 
is incorporated in an analysis of fault-tolerant computer 
architectures assuming that multiple version software is a- 
vailable. Petri net-like models are investigated as a tool 
for performing integrated hardware/software system reliabil- 


ity and performance studies. 


Objectives 


Since software does not fail in the same manner as hard- 
ware, it is difficult to make reasonable tradeoffs between 
hardware fault tolerance and software reliability. Since 
software faults are due to design and implementation errors, 
and not to equipment breakdown, an accepted measure of soft- 


ware reliability has not evolved [1, 2, 3]. A considerable 
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amount of empirical data is available [2, 4, 5, 6], however, 
which quantifies error rates which have been recorded in 
existing systems. Based on this data, it is possible to make 
some reasonable error rate predictions for software developed 
under known conditions. A range for the expected value of 
error rate, therefore, can be determined prior to the develop- 
ment of a software system. 

A number of fault tolerant computer architectures claim 
very high reliability figures with failure probabilities as 
low as io per hour [7, 8]. Such figures are clearly out of 
line with those obtained for software error rates and there- 
fore, represent a false measure of overall system reliability. 

Robert Bernhard (9) identifies software as the key prob- 
lem in achieving a 'no-~downtime' avionic computer which does 
not reduce the overall reliability of the aircraft. Up to 
100,000 lines of code are needed for avionics, and the Fault- 
Tolerant Multiprocessor, Software Implemented Fault Tolerance, 
and other ultrareliable avionic computer designs assume that 
no bugs are present in the software. A close analysis of the 


level of software quality which is achievable is warranted. 
Integrated Hardware/Software Reliability Model 


Several ultrareliable avionic computer architectures 
are currently under development. Of special interest are the 
FTMP (8), SIFT (7), and FTSC (12). A careful look at these 
computer architectures shows that each is designed to prevent 


hardware failures from causing a system failure, but neither 
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directly provides for software redundancy (13). Since either 
a system hardware failure or a system software failure results 


in a system failure, it is apparent that from a total system 
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reliability analysis viewpoint, the hardware and software are 
in series for these systems. Either a hardware or a software 
failure will cause a system failure. The design specifica- 
tions for these systems provide a hardware failure rate of 
107? per hour and assumes perfect software. Accordingly, the 
corresponding system reliability graph is shown below. 


System Hardware System Software 
Reliability Reliability 


fort eis 


* pee 


R(t) = 1-107? R(t) = 1 


The question which is immediately raisec here is: "Is 
R(t) = 1 achievable for approximately 100,000 lines of avi- 
Onic system software, and if not, what is achievable?" 

Although software does not "fail" in the sense of hard- 
ware breakage or malfunction, it does run in real-time so 
that the rate of occurrence of system malfunctions due to 
software "bugs" or imbedded design and implementation errors 
can be determined over an operational time period if suffi- 
cient data is gathered. It is possible therefore, to Speak 
of software-caused system malfunctions in terms of equivalent 
hardware failures. The term "Software Reliability" is used 
in this work in this manner ... as the reliability figure 
which would have been arrived at had the failures been caused 


by hardware malfunction. This hardware/software failure 


equivalence is not recognized by many researchers, but is 


valid from the total system performance viewpoint being 


investigated here. In fact, it is necessary to consider soft- 


ware when determining total system reliability. Assumptions 
and approximations which allow computation of hardware and 
software reliability using the same measure of reliability 
for both are necessary for achieving an optimal total system 
design. 

Proposed methods of determining a numerical value for 
software reliability have fallen into two general categories 
(2), time domain and data domain approaches. The data domain 
approaches. The data domain is quitc important from the 
software analysis and testing viewpoint in that testing of 
various combinations of inputs and tracing through different 
paths through a program are our most effective software de- 
bugging and validation processes. The time domain analysis 
of software failure data is a form of system quality assur- 
ance which is independent of the system structure and of the 
programmer's judgement, therefore, it provides a form of in- 
dependent review of the software quality. From that stand- 
point alone, time domain software quality measures are impor- 
tant, but they also have the advantage of being in a form 
Similar to hardware reliability. This allows a reasonable, 
if not esthetically pleasing, approach to projecting the over- 


all system failure rate. 


It is necessary to make some assumptions in order to have 


a basis for calculating a ‘reliaviilty ‘figure for so.iware. 
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For the purpose of this analysis, it will be assumed that each 


line of code iS a unit and that each unit is equally likely to 
fail in a given time period. Obviously, certain segments of a 
software system will be better designed, more thoroughly tes- 
ted, less complicated, and less failure prone than other seg- 
ments. From an overall system viewpoint, however, the assump- 
tion of equally reliable lines of code all operating in series 
from a reliability analysis viewpoint, is both workable and 
reasonable. The mathematics of hardware reliability may be 
directly applied to software if this assumption is made, and 
Statistics from existing software indicates that longer pro- 
grams tend to have more errors. 

Perhaps the most troublesome issue involved in treating 
hardware and software failures as equivalent events is the 
fact that software errors do not recur once they are repaired. 
Hardware failures are related to wearout and tend to repeat 
themselves on identical units. It is not necessary to become 
involved in failure modes to count failures, however, The 
fact that software failures are unique and could theoretically 
be reduced to zero additional failures after all problems have 
been corrected, whereas, hardware problems recur, may be ig- 
nored during any finite period of the system's life simply by 
putting the two failure phenomena on an equivalent basis, 
namely failures per hour during a portion of the steady-state 
working life of the system. 

Some definitions follow which allow software failure 
probability to be modeled in a manner similar to hardware 


reliability modeling. 
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F(t) Probability Distribution Function 


= Probability that a given line of code will have 
failed by time ‘'t' 


_ Number of failed lines* prior to time t 


Total number of lines 


R(t) Reliability or Probability of success 


= Probability that a given line of code will not 
have failed by time 't'. 


Number of non-failed lines* prior to time t. 
Total number of lines 


f(t) = Failure Density Function 


—. Number of failed lines/Number of hours 


Total number of lines 


Number of never failed lines (number of survivors) 
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= Total number of lines ~ Number of failures 


N = Total number of lines 


The failure density function will be assumed to be expo- 
nential. This corresponds to the decrease in the failure 
rate experienced aS errors are removed from new software. 
Also, an equivalent hazard rate, z(t) = »4, is chosen and set 
equal to a constant, r». The following equations which were 


derived for hardware reliability (Shooman) result: 
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A line which fails, is repaired, then fails again would 
count as two failed lines, so that the F(t) numerator is sim- 
ply the total number of failures prior to time, t. 


-rt 


R(t) =e 


Although it may be argued that the hazard rate has no 
physical meaning for software since the number of surviving 


lines remains constant after repair, it is quite useful as a 


psuedo-hazard rate which serves as a parameter to establish 
the time constant for f(t). Note that for R(t)s1, z(t)sf(t). 
This will be the case for good software during the operations 


phase of its life. 
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it is clear that this model implies that every line of code 
is expected to eventually fail, just as every hardware compo- 
nent is expected to eventually fail. Data from several 
sources (2, 4, 5) indicate that the total number of failures 
expected over the life of a software system is only a small 
fraction of the total number of lines of code in the system. 
Accordingly, an appropriate form for the failure density 


funetion is: 
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f(t) = K, J re" aT + Ky + 8 (#) 
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dirac delta function 


5(t) 


d(C) 


impulse at t = 
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This form for the failure density function incorporates the 
idea that many lines will never fail, that is, they will fail 
at time = ». Schick and Wolverton (2) recorded approximately 


1200 errors in 65,000 lines of code. This suggests a value 

of Ky of .018 and a value of Ky of .982. The software studied 
by Schick and Wolverton did not have such a simple form for 
its failure density curve, but this approach suggests a fail- 
ure density curve of the form shown in Figure 1. 

A range of values for z(t) which corresponds to good to 
extremely high quality software is needed. Schick and Wolverton 
(2) report an error rate during the operations phase of a large 
software system of .025 errors per 1000 lines of code per week. 
This corresponds to .1488 x 10° errors/line/hour. The Bell 
Telephone System's Electronic Switching System has proved to 
have one of the most reliable software packages for which 
recorded data is available. It was designed to have only one 
and one-half hours of downtime in forty years. Assuming that 
this represents a single failure in forty years for 65,000 


lines of code, we get a rough upper bound on software quality 


based on previous data of: 


1 failure 


Failure Rate = (40 years)(365 days/year)(24 hours/day)(65,000 lines 


~11 
4 x 10 failures/line/hour 


Note that the upper bound on system failure rate for the ESS 
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system under these assumptions is: 


1 Failure 


Failure Rate = 40 years 5 days/year 4 hrs/day ) 


_ 2.8 xX 107° failures 
hr./65000 lines. 


This does not approach 107? failures/hour. 


Integrated Hardware/Software Functional Model 


An investigation into the role of Petri net-like models 
as tools to aid in design and analysis of high reliability 
systems (14) indicated that a great deal of information can 
be obtained by using such graphs for modeling and simulation 
of hardware/software systems. The ability of such models to 
reflect the performance of both hardware and softward make 
them a prime candidate for use aS a common language between 
hardware and software professionals. It was clearly deter- 
mined that co-ordination of parallel events such as must occur 
when multiple processors and systems incorporating redundant 
hardware or software are simulated or analyzed can be handled 
by Petri net-like models. 

A Graphic Model of Behavior (14) simulation model of a 


microprocessor was developed early in this project. This 


model simulates the timing and functions of the hardware during 


the execution of assembly language instructions, and as such 


represents a subgraph of the software model. This model in- 


cludes the information present on a hardware timing diagram and 


incorporates a data graph which represents the data on the 


microprocessor pins. A similar model could be obtained for any 
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computer system. 

The results in this study are based on the time domain 
software reliability model developed in the previous section, 
but a few conclusions regarding Petri net-like models may be 
drawn. Such graphs are more useful for simulation and perform- 
ance analysis than for the reliability analysis of existing 
hardware/software systems. They could also be useful in op- 
timizing the reliability of a system being designed if ~intro- 
duced prior to the choice of the hardware/software boundary. 
Their application to complete system modeling (hardware and 
software) is quite feasible, but results in a very detailed 


Simulation. Levels of abstraction are needed. 


Effects of Incorporating Software Redundancy 


Assuming that extremely high reliability is required for 
a system, two alternatives present themselves: 
1. Use modern software design, development, and vali- 
dation procedures to improve the per line failure 
rate of software (15) 


2. Incorporate redundancy in the software system then 
improve the overall system reliability. 


Since additional experience is required to quantify the ef- 
fects of making maximum use of modern software development 
tools, the effects of software redundancy are investigated 
here. Obviously, such an alternative is cost effective only 
if the penalty for a single software failure is extremely high. 
To evaluate the general range of software reliability 
which could be expected from a non-redundant and from a triple- 
modular-redundant software avionic system, a hypothetical 


100,000 line system was broken into modules which ranged in 
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length from one line per module to 10,000 lines per module. 
The psuedo-hazard rate for each line was varied from 9.52 x 
10~° failures/line/hour to 1.16 x oi failures/line/hour, 
and the reliability of each module was calculated by assuming 
that all lines are in series from a reliability analysis 
Standpoint. For an N line module which has hazard rate of 7 
errors/line per hour, the hazard rate for the module is N:-Z, 


-N°Zt 


and the reliability of the module is R(t) =e If three 


software modules are designed and developed for the same soft- 
ware specification, then connected to a 2-out-of-3 voter, the 
reliability (10) of the THR module is R7(3-2R). The overall 
system reliability is obtained by treating all modules as if 
they were in series from a reliability analysis viewpoint. 

The system reliability is therefore the reliability of one 
module, whether redundant or not, raised to an integer power 
equal to the number of modules. These equations appear in the 
Fortran programs in Figures 7 and 8. The variables in the pro- 
gram are defined in Table 19. 

Figure 2 shows the probability of failure as a function 
of the failure rate per line for a hypothetical system which 
consists of 100,000 lines of non-redundant code. The non-re- 
dundant curve shows that even with per line reliability equiv- 


alent to the ESS system (= 10712 


failures/line/hour), the sys- 
tem reliability is only approximately 107° failures per hour. 
This, of course, is far below the quality of ultra-reliable 
avionic computer hardware (107? failures/hour). Clearly, 


either improved software quality on a per line basis or soft- 


ware redundancy is required to achieve software system proba- 
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bility of failure in the neighborhood of 107? /nour. 

The other three curves represent total software system 
reliability assuming that each module is triple redundant. 
Various module lengths are used, but the 100 line/module curve 
is near the appropriate length for most software modules. It 
would probably prove more practical to design redundancy into 
software on the basis of much larger system subsets, however. 
The range of per line reliability required to achieve a total 
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software system reliability in the neighborhood of 10 ~ fail-~ 


ures/hour is seen to be from approximately 107? to 1078 fail- 
ures/line/hour. This level of software quality has already 


been achieved, therefore, one conclusion which can be drawn 


from this graph is that an overall software system reliability 
of 107” failures/hour can be achieved if the cost of develop- 
ing triple modular redundant software can be accepted, and if 
the design specification from which all three independent 
software teams work can be assumed to be perfect. This last 
assumption is a little difficult to defend. 


It may also be observed from Figure 2 that the triple- 
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modular-redundant system reliability improves as the length of 
each module is decreased. This result is analogous to the 
higher system reliability (Shooman) achieved by component re- 
dundancy versus system redundancy for hardware systems. It 

is certainly absurd to consider software redundancy on a per 
line basis, however, even though this appears to be the opti- 
mum strategy according to Figure 2. The reasonable conclusion 
is that if redundancy is employed for software, the length of 


Bi the code in a redundant unit should be as short as possible 
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consistent with retaining the independence of the software 
design and development teams. Redundant units with lengths 
between 100 lines and 10,000 lines, depending on the system, 
appear useful. 

Figures 3 and 4 show the reliability of a single module 
with no software redundancy and with TMR respectively. The 
obvious information here is that module reliability improves 
as module length decreases. The quantitative information 


shows that, for instance, a 10,000 line module (or complete 
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program) with a failure/line/hour of 10 ~ will result in 10. 
failures per line per hour for the module with no redundancy. 
For an equivalent TMR system with the same failure/line/hour, 
a 10,000 line TMR module is seen to exhibit better than a 107? 
system reliability. For a 10,000 line system with system, not 
module, redundancy, this would meet the avionic hardware re- 
liability standards. Figure 5 also shows that module length 
does not affect system reliability for non-redundant code 
under the assumptions of this study. That it is impossible 
to achieve high quality software with excessively long or ex- 
cesSively short modules has been amply documented elsewhere. 
Figure 6 shows that even for non-redundant software, a long 
module is less reliable than a short module, simply because 
each line of code adds another component which may fail. 
Figures 5 and 6 repeat the information in Figures 2, 3, 
and 4 in a different form. The independent variable here is 
module length and the various curves represent various per 


line failure rates. Once again, it is seen that ignoring the 


very real problem of maintaining program team independence 
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when redundant code segments become short, many short TMR 


modules result in higher system reliability than a few long 


TMR modules. 


Conclusions 


1. Hardware and software are in series for reliability anal- 
ysis purposes. 


2. software has been assumed to have R=1 for ultra reliable 
digital system design. 


3. Time domain software measures are independent of the pro- 
grammer's judgement, therefore are a form of independent 
review and are valid as a software quality measure. 


4. Although time domain software measures are difficult to 
apply as predictors of reliability for a particular sys- 
tem, they are quite useful for an assessment of what has 
been achieved and what can be achieved when we look back 
at data gathered over the life span of existing systems. 


9. The assumption that each line of code in a software sys- 
tem is equally likely to fail has some reliability model- 
ing validity from both the evidence of empirical data and 
from the viewpoint of abstraction of stochastic events. 


6. Hardware and software failures may be treated as equiva- 
lent from a reliability viewpoint if the basis for com- 
puting reliability is a simple count of all system fail- 
ures. 


7. A credible failure density function for a single line of 
software is a weighted exponential with a weighted impulse 
at t=~. The impulse at infinity will carry most of the 
failure probability. 


8. The range of failures/line/hour for previous software in- 
cludes at least the range of 10-7 to 10711, 


9. The upper limit of software reliability which can be 
achieved using modern software design and development 
methodology remains to be determined. 


10. Even the best software nreviously developed, may not 
measure up to the 10-“ failures/hour needed for avionic 
systems. 


11. Triple Modular Redundant software, though more expensive, 


can yield system reliability in the 10°” range with soft- 
ware which is no better on a per-line basis than that 
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which has been previously documented. 


Redundancy involving many small software modules yields 
higher reliability than redundancy involving a few large 
modules. This effect is limited by the inability to 
retain design independence for various software teams 
when the size of redundant modules becomes small. 
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Table 19. Variables used in Fortran Programs 
Listed in Figures 7 and 8. 


Variable Definition 

ELAM1 Number of software Failures/Line/Hour 

ALINES Number of lines/module 

ELAMDA Number of software failures/module/hour 

PROB Reliability of one non-redundant module 

R1iMOD Reliability of one TMR module 

ANUMMD Number of modules in the program 

RNONRD Reliability of non-redundant system 

RSYS Reliability of redundant system 

SERRMO Expected number of software errors per 
month 

Pl Probability of Failure of one non- 
redundant module 

R1 Probability of Failure of one TMR 
module 

RS Probability of Failure of redundant 
System 

RN Probability of Failure of non-redundant 
system 
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